Image processing apparatus, image capturing apparatus, and storage medium storing image processing program

ABSTRACT

An image processing apparatus for processing images captured by an image capturing apparatus includes an image input unit to receive a plurality of image frames of a subject captured by the image capturing apparatus along the time line; a tracking area setting unit to set a target tracking area on the plurality of image frames; a tracking unit to track the target tracking area on the plurality image frames; a subject condition prediction unit to predict a condition of the subject at a time point delayed from a time point when the most recent image frame of the plurality of image frames is captured based on a tracking result of the target tracking area; and a parameter setting unit to set a parameter to be used for an image capturing of the subject based on the predicted condition of the subject to the image capturing apparatus.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority pursuant to 35 U.S.C. §119(a) toJapanese Patent Application No. 2015-015424, filed on Jan. 29, 2015 inthe Japan Patent Office, the disclosure of which is incorporated byreference herein in its entirety.

BACKGROUND

Technical Field

The present invention relates to an image processing apparatus, an imagecapturing apparatus, and a storage medium storing an image processingprogram.

Background Art

Image capturing apparatuses have been employed to track moving subjectsor objects, in which the image capturing apparatus continuously capturesa plurality of image frames along the time line to track a movingsubject, and performs the focus and exposure control for capturingimages of the moving subject based on a tracking result.

For example, a camera system applying a template matching process toenhance detection precision of a tracking target is known. Specifically,the camera system corrects brightness of a template image indicating afeature of a subject (tracking target) based on a change of brightnessbetween an area of current position of the tracking target and an areaof predicted movement position of the tracking target in the imageframes.

A description is given of a scheme of a subject tracking with referenceto FIG. 9, in which a train moving with a high speed is tracked as asubject. When a digital camera is used for capturing an image, a shutteris half-pressed to set the focus and exposure level, and then theshutter is released to capture one or more images.

At a timing of FIG. 9A, a user half-presses the shutter to set a targettracking area of the subject as indicated by a rectangular frame F0.Then, the camera performs the subject tracking and obtains a trackingresult indicated by a rectangular frame F1 as shown in FIG. 9B. Thefocus and the exposure level are set based on the tracking result ofFIG. 9B, and then the user releases the shutter of the camera.

However, since the camera requires a given time period to activate theimage capturing process, the subject moves from a position of FIG. 9B toanother position in the given time period. Therefore, when the cameracaptures an image of the subject actually, the subject changes itsposition from the position of FIG. 9B to the position of FIG. 9C, whichmeans the position of the subject of FIG. 9C (rectangular frame F2)deviates from the position of the subject of FIG. 9B (rectangular frameF1) used for setting the focus and the exposure level, which means adelay of the image capturing (i.e., time lag) occurs between FIGS. 9Band 9C. Therefore, an image of the subject in an actually captured imageis out of the focus, and thereby the image of the subject blurs.Further, the exposure level of FIG. 9C also changes from the exposurelevel of FIG. 9B, and thereby the image is captured actually withoutsetting the correct exposure level.

When a moving subject is captured by a camera, a position of the subjectat a moment of the image capturing is required to be measured correctly,and the focus and the exposure level are also required to be setcorrectly. However, the focus and the exposure level of the subject maychange from a time point when the shutter of the camera is half-pressedto a time point when the shutter of the camera is released.

SUMMARY

As one aspect of the present invention, an image processing apparatusfor processing images captured by an image capturing apparatus having anoptical unit is devised. The image processing apparatus includes animage input unit to receive a plurality of image frames of a subjectcaptured by the image capturing apparatus along the time line from theimage capturing apparatus; a tracking area setting unit to set a targettracking area on the plurality of image frames; a tracking unit to trackthe target tracking area on the plurality image frames along the timeline: a subject condition prediction unit to predict a condition of thesubject at a time point delayed from a time point when the most recentimage frame of the plurality of image frames is captured based on atracking result of the target tracking area being tracked along theplurality of image frames; and a parameter setting unit to set aparameter to be used for an image capturing of the subject based on thecondition of the subject predicted by the subject condition predictionunit to the image capturing apparatus.

As another aspect of the present invention, an image capturing apparatusis devised. The image capturing apparatus includes an image capture unitincluding an optical unit to capture and output a plurality of imageframes of the subject along the time line; an image input unit toreceive a plurality of image frames of the subject captured by the imagecapture unit; a tracking area setting unit to set a target tracking areaon the plurality of image frames; a tracking unit to track the targettracking area on the plurality image frames along the time line: asubject condition prediction unit to predict a condition of the subjectat a time point delayed from a time point when the most recent imageframe of the plurality of image frames is captured based on a trackingresult of the target tracking area being tracked along the plurality ofimage frames; and a parameter setting unit to set a parameter to be usedfor an image capturing of the subject to the image capture unit based onthe condition of the subject predicted by the subject conditionprediction unit.

As another aspect of the present invention, a non-transitory storagemedium storing a program that, when executed by a computer, causes thecomputer to execute a method of processing a plurality of image framesof a subject captured by an image capturing apparatus is devised. Themethod includes inputting the plurality of image frames of the subjectcaptured by the image capturing apparatus along the time line; setting atarget tracking area on the plurality of image frames input at theinputting step; tracking the target tracking area, set at the settingstep, on the plurality of image frames along the time line; predicting acondition of the subject at a time point delayed from a time point whenthe most recent image frame of the plurality of image frames is capturedalong the time line based on a tracking result of the target trackingarea obtained at the tracking step; and setting a parameter to be usedfor an image capturing of the subject based on the condition of thesubject predicted at the predicting step to the image capturingapparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1 is an example of a block diagram of a hardware configuration ofan image capturing apparatus according to one or more exampleembodiments of the present invention;

FIG. 2 is a perspective view of the image capturing apparatus of FIG. 1;

FIG. 3 is a functional block diagram of an image processing apparatusaccording to one or more example embodiments of the present invention;

FIG. 4 schematically illustrates an image frame set with a targettracking area processable by the image processing apparatus;

FIG. 5 is a process of tracking a subject performable by using aparticle filter;

FIG. 6 is an example of a profile of tracking results obtained by theimage processing apparatus;

FIG. 7 illustrates a conjugate relationship of an optical unit of theimage capturing apparatus;

FIG. 8 is a flow chart showing the steps of a process of tracking asubject and predicting a movement of the subject implementable by theimage processing apparatus; and

FIGS. 9A, 9B and 9C schematically illustrate image frames set withtarget tracking areas used for a subject tracking implementable by aconventional image processing apparatus.

The accompanying drawings are intended to depict exemplary embodimentsof the present invention and should not be interpreted to limit thescope thereof. The accompanying drawings are not to be considered asdrawn to scale unless explicitly noted, and identical or similarreference numerals designate identical or similar components throughoutthe several views.

DETAILED DESCRIPTION

A description is now given of exemplary embodiments of the presentinvention. It should be noted that although such terms as first, second,etc. may be used herein to describe various elements, components,regions, layers and/or sections, it should be understood that suchelements, components, regions, layers and/or sections are not limitedthereby because such terms are relative, that is, used only todistinguish one element, component, region, layer or section fromanother region, layer or section. Thus, for example, a first element,component, region, layer or section discussed below could be termed asecond element, component, region, layer or section without departingfrom the teachings of the present invention.

In addition, it should be noted that the terminology used herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of the present invention. Thus, for example, asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Moreover, the terms “includes” and/or “including”, when usedin this specification, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof.

Furthermore, although in describing views illustrated in the drawings,specific terminology is employed for the sake of clarity, the presentdisclosure is not limited to the specific terminology so selected and itis to be understood that each specific element includes all technicalequivalents that operate in a similar manner and achieve a similarresult. Referring now to the drawings, one or more image processingapparatuses according to one or more example embodiments are describedhereinafter.

A description is given of an image processing apparatus 1 according toone or more example embodiments of the present invention with referenceto drawings.

FIG. 1 is an example of a block diagram of a hardware configuration ofan image capturing apparatus 30. As illustrated in FIG. 1, the imagecapturing apparatus 30 that can capture images of a subject includes,for example, an optical unit 11, a shutter 12, a charge coupled device(CCD) 13, a correlated double sampling (CDS) circuit 14, ananalog/digital (A/D) converter 15, a motor driver 16, and a timingsignal generator 17. The image capturing apparatus 30 further includes,for example, an image processing circuit 18, a central processing unit(CPU) 19, a random access memory (RANI) 20, a read only memory (ROM) 21,a synchronous dynamic random access memory (SDRAM) 22, acompression/expansion circuit 23, a memory card 24, an operation unit25, and a liquid crystal display (LCD) 26. As described in thisspecification, an image processing according to one or more exampleembodiments of the present invention can be performed by an imageprocessing apparatus 1 shown in FIG. 3. The image capturing apparatus 30is an example of the image processing apparatus 1, in which the imagecapturing and the image processing is performable by the image capturingapparatus 30. Further, the image processing can be performed by using animage capturing device and an image processing apparatus, in which theimage capturing is performed by the image capturing device, and theimage processing is performed by the image processing apparatus.

The optical unit 11 includes at least one or more lenses used forcapturing an image of a subject. The shutter 12 such as a mechanicalshutter is disposed between the optical unit 11 and the CCD 13 to blockor pass light coming from the subject and passing through the opticalunit 11 to the CCD 13. The CCD 13, used as an image generator, includesan imaging face, which may be also referred to an image forming face oran image generation face. The CCD 13 converts an optical image formed orgenerated on the imaging face (image forming or generation face) toanalog image signals. At least the optical unit 11, the shutter 12, andthe charge coupled device (CCD) 13 can be collectively used as an imagecapture unit to capture an image of a subject. For example, the imagecapture unit can capture and output a plurality of image frames of thesubject along the time line.

The CDS circuit 14 removes nose components from the analog image signalsconverted by the CCD 13. The A/D converter 15 converts the analog imagesignals that has removed the nose components by the CDS circuit 14 todigital image signals such as image data.

Under the instruction of the CPU 19, the motor driver 16 drives theoptical unit 11 and the shutter 12 to change positions of the opticalunit 11 and to open and close the shutter 12.

Under the instruction of the CPU 19, the timing signal generator 17generates timing signals to activate the CCD 13, the CDS circuit 14, andthe A/D converter 15. Further, the image processing circuit 18, thecompression/expansion circuit 23, and the memory card 24 can becontrolled by the CPU 19.

The image processing circuit 18 temporarily stores image data outputfrom the A/D converter 15 in the SDRAM 22, and performs various imageprocessing to the image data. The various image processing includes, forexample, YCrCb conversion process, white balance control process,contrast process, edge enhancement process, and color conversionprocess. Further, the image processing circuit 18 instructs to displayan image corresponding the image data having received various imageprocessing on the LCD 26.

The white balance control process is a process of adjusting density ofcolors of image corresponding to image data. The contrast process is aprocess of adjusting contrast of image corresponding to image data. Theedge enhancement process is a process of adjusting sharpness of imagecorresponding to image data. The color conversion process a process ofadjusting color tone of image corresponding to image data.

The ROM 21 stores programs used for executing the subject tracking andsubject movement prediction to be described later. The CPU 19 executesprograms stored in the ROM 21 using the RAM 20 as a working memory.

Based on an instruction input via the operation unit 25, thecompression/expansion circuit 23 compresses the image data havingreceived various image processing by the image processing circuit 18,and stores the compressed image data in the memory card 24. Further, thecompression/expansion circuit 23 expands the image data read from thememory card 24 and outputs the image data to the image processingcircuit 18.

The operation unit 25 can be configured with buttons, switches, levers,and a touch panel. For example, the touch panel is disposed on the LCD26. As illustrated in FIG. 2, the image capturing apparatus 30 includesbuttons such as a power button 31 and a shutter button 32 at the top atthe rear of the image capturing apparatus 30, and the LCD 26 at the rearof the image capturing apparatus 30, in which the power button 31 andthe shutter button 32 can be used as the operation unit 25.

The shutter button 32 is a two-step switch. When a user half-presses theshutter button 32, a half-press signal is transmitted to the CPU 19, andwhen the user presses the shutter button 32 all the way, a full-presssignal is transmitted to the CPU 19.

The LCD 26 can be configured with a liquid crystal panel that candisplay color images, and a touch panel. The LCD 26 can be used as animage display panel to display captured images, and a user interfacepanel to set various settings using the LCD 26. The touch panel of theLCD 26 can receive tapping and drag operation by a user. Further, theLCD 26 can display through images for monitoring with a frame rate suchas one frame per one-thirtieth second (1/30 second). Further, the LCD 26can be used as a finder of checking the angle of view, and a targettracking area setting unit to set a target tracking area to be describedlater.

The image processing program executable by the image processingapparatus 1 can be used to configure modules for the subject tracking,and movement prediction to be described later. The image processingprogram executable by the image processing apparatus 1 can be executedby a hardware. Specifically, the CPU 19 reads the image processingprogram from the ROM 21 used as a storage medium, and executes the imageprocessing program, with which each of units shown in FIG. 3 can beimplemented, and the subject tracking and the movement prediction can beperformed.

FIG. 3 is a functional block diagram of the image processing apparatus1. The image processing apparatus 1 includes, for example, a movie inputunit 101, a target tracking area setting unit 102, a tracking unit 103,a tracking-processed area size prediction unit 104, a subject distanceprediction unit 105, a parameter setting unit 106, a capturingcontroller 107, and a captured image outputting unit 108.

(Input of Movie Image)

The movie input unit 101, used as an image input unit, can be configuredby the CPU 19 and the image processing circuit 18. When the imagecapturing apparatus 30 captures a plurality of image frames such as amovie image or a plurality of still images continuously along the timeline by using the optical unit 11, the shutter 12, the charge coupleddevice (CCD) 13 and others, data of the plurality of image frames aresequentially input to the movie input unit 101. For example, movie imagedata such as through images output from the image processing circuit 18and displayed on the LCD 26 of the image capturing apparatus 30 can beinput to the movie input unit 101.

(Setting of Target Tracking Area)

The target tracking area setting unit 102 can be configured by the CPU19, the operation unit 25, and the LCD 26. The target tracking areasetting unit 102 sets a target tracking area including a focus positionof a subject on one or more image frames input from the movie input unit101. For example, FIG. 4 illustrates a target tracking area indicated bya rectangular frame F, which is shown by a bold line, in which thecenter of the target tracking area is set as the center of the focusposition of the subject. Further, a size of the rectangular frame Findicates a size of the target tracking area. A position of the targettracking area (subject position) and size of the target tracking area(subject size) can be variably set by operating one or more buttons ofthe operation unit 25 and the touch panel of the LCD 26 while theshutter button 32 is being half-pressed by a user.

Further, the target tracking area setting unit 102 stores parameters setfor the target tracking area as a subject condition “S” (x, y, Vx, Vy,Hx, Hy, M) in the RAM 20. As to the subject condition “S” (x, y, Vx, Vy,Hx, Hy, M), “x” and “y” indicate a position of the target tracking area(subject position) indicated by the rectangular frame F of FIG. 4, “Hx”and “Hy” respectively indicate a horizontal size and a vertical size ofthe target tracking area, “Vx” and “Vy” respectively indicate ahorizontal-direction moving speed and a vertical-direction moving speedof the target tracking area, wherein the initial values of “Vx” and “Vy”are set zero, and “M” indicates a size change ratio of the targettracking area (change ratio of size), which is a change ratio of a sizeof the target tracking area between a previous image frame and a currentimage frame, wherein the initial value of “M” is set zero.

(Subject Tracking)

The tracking unit 103 can be configured by the CPU 19. The tracking unit103 tracks a target tracking area from a previous image frame (previousobservation) to a current image frame (current observation) by usingParticle Filter. If the Particle Filter is used, even if a condition ofa subject changes with non-linear manner and distribution of particlesis non-Gaussian, the tracking of the subject can be performed withhigher precision. Further, the method of subject tracking is not limitedto the Particle filter, but other methods such as Mean Shift method, andKalman filter method can be employed.

The tracking unit 103 tracks the target tracking area among a pluralityof image frames along the time line to be described hereinafter.Specifically, the processing of the movement prediction, observation,and correction shown in FIG. 5 are repeatedly performed to the targettracking area to predict the condition of the subject.

(1: Prediction of Movement)

The condition prediction of a subject can be performed by using a knowncondition change model indicated by the below formula (1). The conditionchange model used for the Particle filter is non-linear.S _(k) =f _(k)(S _(k−1) ,V _(k−1))  (1)

The formula (1) expresses the condition change from one condition“S_(k−1)” to a next condition “S_(k)” When the movement prediction isperformed, the condition change or transition can be expressed by thebelow formulas (2) to (8).x _(k) =x _(k−1) +V _(xk−1) *Δt  (2)y _(k) =y _(k−1) +V _(yk−1) *Δt  (3)V _(xk) =V _(xk−1)  (4)V _(yk) =V _(yk−1)  (5)M _(k) =M _(k−1)  (6)H _(xk) =H _(xk−1)(1+M _(k−1))  (7)H _(yk) =H _(yk−1)(1+M _(k−1))  (8)

By using the above formulas (2) to (8), the next condition “S_(k)” canbe predicted from the previous condition “S_(k−1).”

(2: Observation)

Then, the condition observation formula (9) is applied to the condition“S” obtained at the movement prediction process to perform theobservation.z _(k) =h _(k)(S _(k),n_(k))  (9)

In the formula (9), “z_(k)” indicates observation data of the condition“S_(k)”. The observation data can be expressed by a color histogram forthe target tracking area (FIG. 4) of the subject. The calculation of thecolor histogram can be performed by using the formula (10).

In the formula (10), it is assumed that the target tracking area iscomposed of a given number of pixels such as “I” pixels, wherein “I” isa natural number. In the formula (10), “xi” indicates a position of the“i-th” pixel, “y” indicates a position of a pixel at the center of thetarget tracking area, “h(xi)” indicates a color pixel value of the“i-th” pixel, and “u” indicates a brightness value such as from 0 to255, in which “i” is a natural number.

In this processing, “k(r)” indicated by the formula (11) is Kernelfunction used for calculating the color histogram. In the formula (10),Kernel function “k(r)” becomes zero when a distance between the pixelposition “y” and the pixel position “xi” is a constant “a” or more, andKernel function k(r) becomes the maximum value when the pixel position“y” matches the pixel position “xi.” With this configuration, the numberof pixels having the brightness “u” in the target tracking area can becalculated while reducing an effect of an area around the targettracking area in the formula (10).

$\begin{matrix}{p_{y}^{(u)} = {f{\sum\limits_{i = 1}^{I}\;{{k\left( \frac{{y - x_{i}}}{a} \right)}{\delta\left\lbrack {{h\left( x_{i} \right)} - u} \right\rbrack}}}}} & (10) \\{{k(r)} = \left\{ \begin{matrix}{1 - r^{2}} & \vdots & {r < 1} \\0 & \vdots & {otherwise}\end{matrix} \right.} & (11)\end{matrix}$

The subject tracking can be performed by performing the movementprediction and observation shown in FIG. 5. In this configuration, thesubject tracking is performed by using “N” number of predictionconditions “Sk(i)” for “N” number of particles (N: natural number),which means the number of particles used for the subject tracking is set“N”. The “N” number of particles are evaluated, and a weight coefficientis calculated for each of the “N” number of particles. Then, a weightedaverage obtained from the calculated weight coefficients of the “N”number of particles indicated by the below formula (12) becomes thecondition of the tracking-processed area.

$\begin{matrix}{S_{k} = {\sum\limits_{i = 1}^{N}\;{\pi^{i}S_{k}^{(i)}}}} & (12)\end{matrix}$

A description is given of calculation of the weight coefficient.Distribution of particles expressed by the formula (12) is set as randomsampling. The condition prediction of each one of the particles can beperformed by using the below formula (1′) and formulas (13) to (19), inwhich each of the formulas (13) to (19) is set by respectively adding arandom variable to the formulas (2) to (8). Terms of “r1” to “r7” in theformulas (13) to (19) are Gaussian random variables.S _(k) ^((i)) =f _(k)(S _(k−1) ^((i)) ,V _(k−1))  (1′)x _(k) =x _(k−1) +V _(xk−1) *Δt+r ₁  (13)y _(k) =y _(k−1) +V _(yk−1) *Δt+r ₂  (14)V _(xk) =V _(xk−1) +r ₃  (15)V _(yk) =V _(yk−1) +r ₄  (16)M _(k) =M _(k−1) +r ₅  (17)H _(xk) =H _(xk−1)(1+M _(k−1))+r ₆  (18)H _(yk) =H _(yk−1)(1+M _(k−1))+r ₇  (19)

The condition “Sk(i)” of each of the particles can be predicted byapplying the formulas (13) to (19). A description is given ofcalculating the weight coefficient “π^(i)” of the predicted condition“Sk(i).” In this process, the color histogram “p” calculated by usingthe formula (10) for a target tracking area of a subject is used as atemplate. Further, the color histogram of each of the predicted “N”number of particles is set as “q,” and Bhattacharyya coefficientexpressed by the formula (20) is calculated. The

Bhattacharyya coefficient indicates a similarity level between the colorhistogram “p” of the target tracking area of the subject and the colorhistogram “q” of one predicted particle. The greater the Bhattacharyyacoefficient, the higher the similarity level of the two colorhistograms. In the formula (20), “m” is the number of the brightness “u”such as 256, which is the maximum number settable for the brightness“u”.

$\begin{matrix}{{\rho\left\lbrack {p,q} \right\rbrack} = {\sum\limits_{u = 1}^{m}\;{\sqrt{p^{(u)}q^{(u)}}.}}} & (20)\end{matrix}$

The weight coefficient “π^(i)” is calculated by using the formula (21)and the Bhattacharyya coefficient.

$\begin{matrix}{\pi^{i} = {{\frac{1}{\sqrt{{2\;\pi}\;}\sigma}{\mathbb{e}}^{- \frac{d^{2}}{2\;\sigma^{2}}}} = {\frac{1}{\sqrt{2\;\pi}\;\sigma}{\mathbb{e}}^{- \frac{({1 - {\rho{\lbrack{p,q}\rbrack}}})}{2\;\sigma^{2}}}}}} & (21) \\{d = \sqrt{1 - {\rho\left\lbrack {p,q} \right\rbrack}}} & (22)\end{matrix}$

Based on the calculated weight coefficient “π^(i)” and the predictedcondition “Sk(i),” the condition of the tracking-processed area appliedwith the Particle filter can be calculated by using the formula (12).

(3: Correction)

When the particles are predicted by using the formula (12), an effect ofparticles having smaller weight coefficients becomes smaller while theeffect of particles having greater weight coefficients becomes greaterin the formula (12), with which the number of effective particlesdecreases. To prevent this decrease, the particles having the smallerweight coefficients are deleted, and the particles having the greaterweight coefficients are re-sampled and duplicated. With this processing,the “N” number of particles can be used effectively.

If a shape of a tracking subject changes gradually during the tracking,a template of the target tracking area of the subject is updated. Inthis process, if the average “π^(ES)” of the weight coefficients of thetracking-processed area calculated by the formula (12) becomes a valuegreater than a threshold “π^(T)” as indicated by the formula (23), thetemplate is updated by using the formula (24), in which “α” is a ratiocoefficient, wherein the value of “α” is set to a smaller value in theformula (24), and the color histogram of each of the particles in thetarget tracking area of the subject is updated.π^(ES)>π^(T)  (23)q _(t) ^((u))=(1−α)q _(t−1) ^((u)) +αp _(t−1) ^((u))  (24)

As described above, the subject tracking is performed by repeating thesequential processing of movement prediction, observation andcorrection. Based on the subject tracking, the condition “S” of thetracking-processed area can be calculated as a tracking result of thetarget tracking area. Further, since a time lag “Δt” of the imagecapturing apparatus 30 can be set as a pre-set value, the subjectcondition at a time point of the time lag “Δt” delayed from a time pointwhen the shutter button 32 is pressed all the way can be predicted basedon the condition at the time point when the shutter button 32 is pressedall the way. The subject condition at a time point of the time lag “Δt”means the position of the tracking-processed area indicated by “x” and“y” and the tracking-processed area size indicated by “Hx and Hy” at atime point corresponding to the time lag “Δt,” Therefore, the imageprocessing apparatus 1 can predict the subject condition on an imageframe at a time point delayed for a given time from the most recentlycaptured image frame along the time line of the plurality of input imageframes.

In this process, the term of “Hx+Hy” is referred to a tracking-processedarea size “X.” The position of the tracking-processed area indicated by“x” and “y” at a time point when an image capturing operation isperformed can be calculated by using the formulas (2) and (3), and thetracking-processed area size “X” at a time point when the imagecapturing operation is performed can be predicted by using the formulas(7) and (8).

However, the size change ratio “M” predicted from the previous imageframe may have some error due to the effect of noise. Therefore, asdescribed below, the tracking-processed area size “X” is predicted byapplying a polynomial approximation to data of the plurality of thetracking-processed area sizes obtained from the tracking results of theplurality of image frames.

(Prediction of Tracking-processed Area Size)

From a time point when the shutter button 32 is half-pressed to a timepoint when the shutter button 32 is pressed all the way, a plurality ofimage frames is captured, and tracking results of the plurality of imageframes can be obtained. The tracking-processed area size prediction unit104 can predict the tracking-processed area size “X” at a time pointcorresponding to the time lag “Δt” as the subject condition by applyingthe polynomial approximation to the tracking results of the plurality ofimage frames. In other words, the tracking-processed area sizeprediction unit 104 can predict the tracking-processed area size “X”based on the condition “S” at a time point that the shutter button 32 ispressed all the way, and a plurality of conditions “S” obtained at timepoints before the time point when the shutter button 32 is pressed allthe way.

A description is given of prediction of the tracking-processed area size“X” at a time point corresponding to the time lag “Δt” by using thetracking-processed area size prediction unit 104.

FIG. 6 is an example of a profile of tracking results obtained by theimage processing apparatus, in which the horizontal axis represents time“T”, and the vertical axis represents the tracking-processed area size“X”. Each of black dots indicates the tracking-processed area size ateach one of image frames before the shutter button 32 is pressed all theway except the last black dot (right-end black dot) in FIG. 6corresponding to a time point when the shutter button 32 is pressed allthe way, while a white dot indicates a prediction size of thetracking-processed area size “X” at a time point corresponding to thetime lag “Δt” delayed from the time point when the shutter button 32 ispressed all the way.

The formula (25) is a polynomial expression used for predicting thetracking-processed area size X at a time point corresponding to the timelag “Δt,” in which “X” is the tracking-processed area size, “a” is acoefficient of polynomial expression, and “t” is time. As indicated bythe formula (25), the tracking-processed area size “X” can be predictedwhen the coefficients of polynomial expression are obtained.X=a ₀ +a ₁ t+a ₂ t ² + . . . +a _(n−1) t ^(n−1)  (25)

Further, the formula (25) can be expressed by a matrix as indicated bythe formula (26).X=H*A  (26)

In this process, the vector “X” and the vector “A” in the formula (26)can be respectively expressed by the formula (27) and the formula (28).As to the formula (27), “X₀” to “X_(m−1)” are data of thetracking-processed area size at “m” points. As to the formula (28), “a₀”to “a_(n−1)” are “n” coefficients set for the polynomial expression.

Further, the matrix “H” in the formula (26) can be expressed by theformula (29).

$\begin{matrix}{X = \begin{bmatrix}X_{0} \\X_{1} \\\vdots \\\vdots \\X_{m - 1}\end{bmatrix}} & (27) \\{A = \begin{bmatrix}a_{0} \\a_{1} \\\vdots \\\vdots \\a_{n - 1}\end{bmatrix}} & (28) \\{H = \begin{bmatrix}1 & t_{0} & t_{0}^{2} & t_{0}^{3} & \ldots & \ldots & t_{0}^{n - 1} \\1 & t_{1} & t_{1}^{2} & t_{1}^{3} & \ldots & \ldots & t_{1}^{n - 1} \\1 & t_{2} & t_{2}^{2} & t_{2}^{3} & \ldots & \ldots & t_{2}^{n - 1} \\1 & t_{3} & t_{3}^{2} & t_{3}^{3} & \ldots & \ldots & t_{3}^{n - 1} \\1 & t_{4} & t_{4}^{2} & t_{4}^{3} & \ldots & \ldots & t_{4}^{n - 1} \\1 & t_{5} & t_{5}^{2} & t_{5}^{3} & \ldots & \ldots & t_{5}^{n - 1} \\\; & \; & \ldots & \; & \; & \; & \ldots \\\; & \; & \ldots & \; & \; & \; & \ldots \\\; & \; & \ldots & \; & \; & \; & \ldots \\1 & t_{m - 1} & t_{m - 1}^{2} & t_{m - 1}^{3} & \ldots & \ldots & t_{m - 1}^{n - 1}\end{bmatrix}} & (29)\end{matrix}$

The coefficients of polynomial expression “a₀” to “a_(n−1)” can becalculated by using the formula (30) and the method of least squares.A=(H ^(T) H)⁻¹ H ^(T) X  (30)

When the coefficients of polynomial expression “a₀” to “a_(n−1)” arecalculated by using the formula (30), the tracking-processed area size“X” at a time point corresponding to the time lag “Δt” can be calculatedby using the formula (25).

(Prediction of Subject Distance)

As illustrated in FIG. 7, the subject distance prediction unit 105 cancalculate the subject distance “u₁” as the subject condition at a timepoint corresponding to the time lag “Δt” when an image capturingoperation is performed actually based on the subject distance “u” at atime point when the shutter button 32 is pressed all the way. Thesubject distance means the distance between the subject and the opticalunit 11 of the image capturing apparatus 30.

A description is given of a method of predicting the subject distance“u₁”at a time point corresponding to the time lag “Δt” based on thetracking-processed area size “X” at a time point corresponding to thetime lag “Δt” predicted by the tracking-processed area size predictionunit 104.

FIG. 7 illustrates a conjugate relationship of the optical unit 11, inwhich the principle of forming an image on an image sensor by using theoptical unit 11 such as one or more lenses or the like is illustrated.In the configuration of FIG. 7, “u, Hv, v” respectively represents“subject distance, subject image height, subject imaging distance” at atime point when the shutter button 32 is pressed all the way while “u₁,Hv₁ 1, v₁” respectively represents “predicted subject distance,predicted subject image height, predicted subject imaging distance” at atime point corresponding to the time lag “Δt,” and “H” represents thesubject height, and “F” represents the focal length of the optical unit11.

As indicated in the configuration of FIG. 7, the conjugate relationshipexpressed by the formulas (31) and (32) can be set based on thesimilarity relationship of the triangle.

$\begin{matrix}{\frac{H}{u} = \frac{H_{v}}{v}} & (31) \\{\frac{H}{u_{1}} = \frac{H_{v\; 1}}{v_{1}}} & (32)\end{matrix}$

Then, the formula (33) can be obtained from the formulas (31) and (32).

$\begin{matrix}{u_{1} = {u\frac{H_{v}}{H_{v\; 1}}\frac{v_{1}}{v}}} & (33)\end{matrix}$

When the formula (33) is substituted in the formulas (34) and (35),which are the lens equations, the formula (36) can be obtained.

$\begin{matrix}{{\frac{1}{u} + \frac{1}{v}} = \frac{1}{F}} & (34) \\{{\frac{1}{u_{1}} + \frac{1}{v_{1}}} = \frac{1}{F}} & (35) \\{u_{1} = {{u\frac{H_{v}}{H_{v\; 1}}} + {F\frac{H_{v\; 1} - H_{v}}{H_{v\; 1}}}}} & (36)\end{matrix}$

The predicted subject distance “u₁” can be calculated by using theformula (36), in which “Hv” is the subject image height on the imagingface of the CCD 13 when the shutter button 32 is pressed all the way,which is a known value, and the focal length “F” is also a known value.The predicted subject image height “Hv₁” on the imaging face of the CCD13 can be calculated from the subject image height “Hv,” thetracking-processed area size “X” when the shutter button 32 is pressedall the way, and the tracking-processed area size “X” at a time pointcorresponding to the time lag “Δt” calculated by using the formula (25)to (30). Therefore, the predicted subject distance “u₁” can becalculated by using the formula (36).

Further, if the focal length “F” is too small compared to the subjectdistance “u,” the formula (36) can be approximated to the formula (37)because the second term of the formula (36) becomes too small comparedto the first term of the formula (36). As indicated by the formula (37),the subject distance before and after the prediction is inverselyproportional to the tracking-processed area size before and after theprediction.

$\begin{matrix}{u_{1} \approx {u\frac{H_{v}}{H_{v\; 1}}}} & (37)\end{matrix}$

Therefore, the predicted subject distance “u₁” can be calculated byusing the formula (36) or the formula (37). If a focus position (lensposition) of the optical unit 11 of the image capturing apparatus 30 isset in view of the predicted subject distance “u₁”, the focusing of thesubject can be set correctly when the image capturing operation isperformed for a subject moving with a high speed even if the time lagoccurs for the image capturing operation.

(Setting of Capturing Parameter)

The parameter setting unit 106 can be configured by the CPU 19. Theparameter setting unit106 sets one or more capturing parameters to thecapturing controller 107 based on data of the subject condition outputfrom the tracking unit 103, the tracking-processed area size predictionunit 104, and the subject distance prediction unit 105. As to thesubject condition, the subject distance prediction unit 105 outputs thesubject distance “u₁” at a time point corresponding to the time lag“Δt,” the tracking-processed area size prediction unit 104 outputs thetracking-processed area size “X” at a time point corresponding to thetime lag “Δt,” and the tracking unit 103 outputs the position of thetracking-processed area indicated by “x” and “y” at a time pointcorresponding to the time lag “Δt.”

The parameter setting unit 106 sets a focus position of the optical unit11 to the capturing controller 107 as a capturing parameter based on thesubject distance “u₁” at a time point corresponding to the time lag“Δt.” Further, the parameter setting unit106 sets an exposure level tothe capturing controller 107 as a capturing parameter based on theposition of the tracking-processed area indicated by “x” and “y” at atime point corresponding to the time lag “Δt,” and thetracking-processed area size “X” at a time point corresponding to thetime lag “Δt.”

The capturing controller 107 can be configured by the CPU 19. Thecapturing controller 107 performs the focus control and the exposurecontrol to the above described predicted tracking-processed area byapplying the above described capturing parameters, and then thecapturing controller 107 captures the subject at a time pointcorresponding to the time lag “Δt” delayed from a time point when theshutter button 32 is pressed all the way.

Further, the capturing controller 107 also performs the focus controland the exposure control to the tracking-processed areas being updatedsequentially while the shutter button 32 is being half-pressed byapplying the capturing parameters. In this process, the capturingparameters can be set based on the subject distance “u,” the position ofthe tracking-processed area indicated by “x” and “y” obtainable by usingthe formulas (2) and (3), and “Hx” and “Hy” indicating thetracking-processed area size obtainable by using the formulas (7) and(8).

The captured image outputting unit 108 can be configured by the LCD 26,the compression/expansion circuit 23, and the memory card 24. Thecaptured image outputting unit 108 compresses the image frames havingreceived the focus control and the exposure control by the capturingcontroller 107, and stores the compressed image frames in the memorycard 24. Further, the captured image outputting unit 108 can display theimage frames having received the focus control and the exposure controlon the LCD 26.

A description is given of a process implementable by the imageprocessing program executable by the image processing apparatus 1 withreference to FIG. 8, which is a flow chart showing the steps of aprocess of tracking a subject and predicting a movement of the subjectimplementable by the image processing apparatus 1. Specifically, when auser half-presses the shutter button 32, the sequence of FIG. 8 starts.

At step S1, the movie input unit 101 is input with image data such as aplurality of image frames of a subject captured along the time line, inwhich the image frames are movie images or a plurality of still imagescaptured by the image capturing apparatus 30 along the time line. Forexample, data of movie image of through images displayed on the LCD 26of the image capturing apparatus 30 are sequentially input.

At step S2, the target tracking area setting unit 102 sets a targettracking area of the subject on an image frame F(k−1).

At step S3, the tracking unit 103 calculates a feature of the targettracking area. For example, the tracking unit 103 calculates a colorhistogram of the target tracking area as the feature by using theformula (10).

At step S4, the tracking unit 103 searches a candidate area of thetracking-processed area on an image frame F(k), which is a next frame ofthe image frame F(k−1), by using the formulas (13) to (19).

At step S5, the tracking unit 103 calculates the feature of thecandidate area of the tracking-processed area on the image frame F(k).For example, a color histogram of the candidate area of thetracking-processed area is calculated as the feature. Further, thesimilarity level between the target tracking area on the image frameF(k−1), and the candidate area of the tracking-processed area on theimage frame F(k) is calculated by using the formula (20) and thefeatures calculated at step S3 and step S5.

At step S6, the tracking unit 103 calculates or determines thetracking-processed area on the image frame F(k) by using the formula(12). At step S7, the tracking unit 103 calculates each of parametersconfiguring the condition “S_(k)” of the tracking-processed area,obtainable by the formula (12), by using the formula (2) to (8). Amongthese parameters, a position of the tracking-processed area on the imageframe F(k) can be obtained based on the position of thetracking-processed area indicated by “x” and “y,” and “Hx” and “Hy”indicating the tracking-processed area size.

At step S8, the capturing controller 107 performs the focus control andthe exposure control to the tracking-processed area based on the subjectdistance “u,” the position of the tracking-processed area indicated by“x” and “y,” and “Hx” and “Hy” indicating the tracking-processed areasize.

At step S9, the capturing controller 107 determines whether the shutterbutton 32 is pressed all the way. If the shutter button 32 is stillbeing half-pressed, the sequence returns to step S4, and the trackingunit 103 searches a candidate area of the tracking-processed area on anew image frame F(k+1). By contrast, if the shutter button 32 is pressedall the way, the sequence proceeds to step S10.

At step S10, the tracking-processed area size prediction unit 104calculates the tracking-processed area size “X” at a time pointcorresponding to the time lag “Δt” by applying the polynomialapproximation using the formula (25) to (30). Further, the subjectdistance prediction unit 105 calculates the predicted subject distance“u₁” at a time point corresponding to the time lag “Δt” by using theformula (36) or formula (37).

At step S11, the tracking unit 103 calculates the position of thetracking-processed area indicated by “x” and “y” at a time pointcorresponding to the time lag “Δt” by using the formulas (2) and (3).

At step S12, the parameter setting unit 106 sets the capturingparameters based on the subject condition obtained at steps S10 and S11to the capturing controller 107. Then, the capturing controller 107performs the focus control and the exposure control to thetracking-processed area at a time point corresponding to the time lag“Δt” based on the set capturing parameters.

Then, at step S13, an image frame is captured, and the captured imageoutputting unit 108 compresses the image frame captured with the focuscontrol and the exposure control set at step S12, and stores thecompressed image frame in the memory card 24. Further, the capturedimage outputting unit 108 displays the image frame on the LCD 26.

In the above described image processing apparatus 1, the subjectcondition at a time point corresponding to the time lag “Δt” that isdelayed from the most recent image frame of the plurality of imageframes captured along the time line can be predicted based on thetracking results of the target tracking area being tracked among theplurality of image frames. In this configuration, the subject conditionincludes the position of the tracking-processed area indicated by “x”and “y,” the tracking-processed area size “X” indicated by “Hx and Hy,”and the subject distance “u₁” at a time point corresponding to the timelag “Δt” delayed from the time point when the shutter is released.Further, the capturing parameters determined from the predicted subjectcondition can be set to the image capturing apparatus 30.

With employing this configuration, even if the time lag occurs from thetime point when the shutter is released to the time point when the imagecapturing is actually performed, the subject position and the subjectdistance can be predicted correctly, and the focus level and theexposure level can be set correctly by performing the movementprediction based on the tracking results of the plurality of imageframes.

Further, as to the above described image processing apparatus 1, thepolynomial approximation can be applied to the tracking-processed areasize “X” of each of the image frames, with which the tracking-processedarea size “X” at a time point corresponding to the time lag “Δt” can bepredicted as the subject condition. With employing this configuration,the tracking-processed area size “X” at a time point corresponding tothe time lag “Δt” can be predicted from the time point that the shutteris released, and the focusing of the subject can be performed correctlywhen to perform the image capturing operation.

Further, as to the above described image processing apparatus 1, thesubject distance “u₁” at a time point corresponding to the time lag “Δt”can be predicted as the subject condition based on the focal length ofthe optical unit 11, the tracking-processed area size “X” at a timepoint corresponding to the time lag “Δt”, and the conjugate relationshipof the optical unit 11. With employing this configuration, the subjectdistance “u₁” at a time point corresponding to the time lag “Δt” delayedfrom the time point that the shutter is released can be predicted, andthe focusing of the subject for the image capturing operation can beperformed correctly.

Further, as to the above described image processing apparatus 1, theparameter setting unit 106 outputs the focus position of the opticalunit 11 based on the predicted subject distance “u₁” as a capturingparameter. With employing this configuration, when to perform the imagecapturing operation at a time point corresponding to the time lag “Δt”delayed from the time point that the shutter is released, the focusingof the subject can be performed correctly.

Further, as to in the above described image processing apparatus 1, theparameter setting unit106 outputs the exposure level based on thepredicted position of the tracking-processed area indicated by “x” and“y” at a time point corresponding to the time lag “Δt,” and thepredicted tracking-processed area size “X” indicated by “Hx and Hy” at atime point corresponding to the time lag “Δt” as a capturing parameter.With employing this configuration, when to perform the image capturingoperation at a time point corresponding to the time lag “Δt” delayedfrom the time point that the shutter is released, the exposure of thesubject can be performed correctly.

The above described image processing apparatus, image capturingapparatus, and image processing program can predict a position of asubject and a distance to the subject when an image capturing operationof the subject is to be performed based on one or more tracking resultsobtainable by tracking the subject before capturing an image of thesubject actually.

The present invention can be implemented in any convenient form, forexample using dedicated hardware platform, or a mixture of dedicatedhardware platform and software. Each of the functions of the describedembodiments may be implemented by one or more processing circuits orcircuitry. Processing circuitry includes a programmed processor, as aprocessor includes circuitry. A processing circuit also includes devicessuch as an application specific integrated circuit (ASIC) andconventional circuit components arranged to perform the recitedfunctions. For example, in some embodiments, any one of the informationprocessing apparatus may include a plurality of computing devices, e.g.,a server cluster, that are configured to communicate with each otherover any type of communication links, including a network, a sharedmemory, etc. to collectively perform the processes disclosed herein.

The computer software can be provided to the programmable device usingany carrier medium or storage medium such as non-volatile memory forstoring processor-readable code such as a floppy disk, a flexible disk,a compact disk read only memory (CD-ROM), a compact disk rewritable(CD-RW), a digital versatile disk read only memory (DVD-ROM), DVDrecording only/rewritable (DVD-R/RW), electrically erasable andprogrammable read only memory (EEPROM), erasable programmable read onlymemory (EPROM), a memory card or stick such as USB memory, a memorychip, a mini disk (MD), a magneto optical disc (MO), magnetic tape, ahard disk in a server, a flash memory, Blu-ray disc (registeredtrademark), secure digital (SD) card, a solid state memory device or thelike, but not limited these. Further, the computer software can beprovided through communication lines such as electrical communicationline. Further, the computer software can be provided in a read onlymemory (ROM) disposed for the computer. The computer software stored inthe storage medium can be installed to the computer and executed toimplement the above described processing. The computer software storedin the storage medium of an external apparatus can be downloaded andinstalled to the computer via a network to implement the above describedprocessing.

The hardware platform includes any desired kind of hardware resourcesincluding, for example, a central processing unit (CPU), a random accessmemory (RAM), and a hard disk drive (HDD). The CPU may be implemented byany desired kind of any desired number of processors. The RAM may beimplemented by any desired kind of volatile or non-volatile memory. TheHDD may be implemented by any desired kind of non-volatile memorycapable of storing a large amount of data. The hardware resources mayadditionally include an input device, an output device, or a networkdevice, depending on the type of apparatus. Alternatively, the HDD maybe provided outside of the apparatus as long as the HDD is accessible.In this example, the CPU, such as a cache memory of the CPU, and the RAMmay function as a physical memory or a primary memory of the apparatus,while the HDD may function as a secondary memory of the apparatus.

In the above-described example embodiment, a computer can be used with acomputer-readable program, described by subject-oriented programminglanguages such as C, C++, C#, Java (registered trademark), JavaScript(registered trademark), Perl, Ruby, or legacy programming languages suchas machine language, assembler language to control functional units usedfor the apparatus or system. For example, a particular computer (e.g.,personal computer, workstation) may control an information processingapparatus or an image processing apparatus using a computer-readableprogram, which can execute the above-described processes or steps. Inthe above-described embodiments, at least one or more of the units ofapparatus can be implemented as hardware or as a combination ofhardware/software combination. Each of the functions of the describedembodiments may be implemented by one or more processing circuits. Aprocessing circuit includes a programmed processor, as a processorincludes circuitry. A processing circuit also includes devices such asan application specific integrated circuit (ASIC) and conventionalcircuit components arranged to perform the recited functions.

Numerous additional modifications and variations for the communicationterminal, information processing system, and information processingmethod, a program to execute the information processing method by acomputer, and a storage or carrier medium of the program are possible inlight of the above teachings. It is therefore to be understood thatwithin the scope of the appended claims, the disclosure of the presentinvention may be practiced otherwise than as specifically describedherein. For example, elements and/or features of different examples andillustrative embodiments may be combined each other and/or substitutedfor each other within the scope of this disclosure and appended claims.

What is claimed is:
 1. An image processing apparatus for processingimages captured by an image capturing apparatus having an optical unit,the image processing apparatus comprising: processing circuitryconfigured to receive a plurality of image frames of a subject capturedby the image capturing apparatus along a time line from the imagecapturing apparatus; set a target tracking area on the plurality ofimage frames; track the target tracking area on the plurality imageframes along the time line between the plurality of image frames thatare captured from a time point when a shutter button is partiallypressed to a time point when the shutter button is fully pressed;predict a condition of the subject at a time point delayed from a timepoint when the most recent image frame of the plurality of image framesis captured based on a tracking result of the target tracking area beingtracked along the plurality of image frames, wherein the condition ofthe subject is a tracking-processed area size; and set a parameter to beused for an image capturing of the subject based on the predictedcondition of the subject, wherein the parameter is an exposure levelthat is set based on a focus position at a subject distance, a positionof the tracking-processed area size, and a size of thetracking-processed area size.
 2. The image processing apparatus of claim1, wherein the processing circuitry predicts a position of the subjecton one image frame at the time point, delayed from the time point whenthe most recent image frame of the plurality of image frames iscaptured, to determine the condition of the subject.
 3. The imageprocessing apparatus of claim 1, wherein the processing circuitryapplies a polynomial approximation to the tracking result to predict asize of the subject on one image frame at the time point, delayed fromthe time point when the most recent image frame of the plurality ofimage frames is captured, to determine the condition of the subject. 4.The image processing apparatus of claim 1, wherein the processingcircuitry predicts a distance to the subject from the optical unit atthe time point, delayed from the time point when the most recent imageframe of the plurality of image frames is captured, to determine thecondition of the subject based on a focal length of the optical unit,the size of the subject predicted on the one image frame, and aconjugate relationship of the optical unit.
 5. The image processingapparatus of claim 4, wherein the processing circuitry outputs a focusposition of the optical unit calculated from the predicted distancebetween the subject and the optical unit as the capturing parameter. 6.The image processing apparatus of claim 2, wherein the processingcircuitry outputs an exposure level calculated from the position and asize of the subject predicted on the one image frame as the capturingparameter.
 7. An image capturing apparatus comprising: an image capturedevice including an optical unit to capture and output a plurality ofimage frames of a subject along a time line; and processing circuitryconfigured to receive the plurality of image frames of the subjectcaptured by the image capture unit; set a target tracking area on theplurality of image frames; track the target tracking area on theplurality image frames along the time line between the plurality ofimage frames that are captured from a time point when a shutter buttonis partially pressed to a time point when the shutter button is fullypressed; predict a condition of the subject at a time point delayed froma time point when the most recent image frame of the plurality of imageframes is captured based on a tracking result of the target trackingarea being tracked along the plurality of image frames, wherein thecondition of the subject is a tracking-processed area size; and set aparameter to be used for an image capturing of the subject to the imagecapture unit based on the predicted condition of the subject, whereinthe parameter is an exposure level that is set based on a focus positionat a subject distance, a position of the tracking-processed area size,and a size of the tracking-processed area size.
 8. A non-transitorystorage medium storing a program that, when executed by a computer,causes the computer to execute a method of processing a plurality ofimage frames of a subject captured by an image capturing apparatus, themethod comprising: inputting the plurality of image frames of thesubject captured by the image capturing apparatus along a time line;setting a target tracking area on the plurality of image frames input atthe inputting step; tracking the target tracking area, set at thesetting step, on the plurality of image frames along the time linebetween the plurality of image frames that are captured from a timepoint when a shutter button is partially pressed to a time point whenthe shutter button is fully pressed; predicting a condition of thesubject at a time point delayed from a time point when the most recentimage frame of the plurality of image frames is captured along the timeline based on a tracking result of the target tracking area obtained atthe tracking step, wherein the condition of the subject is atracking-processed area size; and setting a parameter to be used for animage capturing of the subject based on the condition of the subjectpredicted at the predicting step to the image capturing apparatus,wherein the parameter is an exposure level that is set based on a focusposition at a subject distance, a position of the tracking-processedarea size, and a size of the tracking-processed area size.